Adding Synchronization and Rolling Shutter in Multi-Camera Bundle Adjustment

This paper introduces a new bundle adjustment for multi-cameras, that simultaneously estimates not only the usual parameters (poses and points) but also the synchronization and the rolling shutter. We start from an initial calibration with a global shutter camera model (GS) and a frameaccurate synchronization provided by previous self-calibration methods [1]. Our BA provides subframe-accurate synchronization (SFA), i.e. it estimates the residual time offsets ∆ j between a reference camera and the others. It also estimate the rolling shutter (RS) coefficient, i.e. the time delay τ between two adjacent lines of a frame. We present a continuous camera trajectory model that provides the multi-camera pose at every time. Let R be a C1 continuous function that maps Ω ⊆ Rk to rotation set SO(3). The camera trajectory is modeled by a C3 continuous function M : [0,1]→ R3×Ω such that M(t)T = ( TM(t) EM(t) ) and TM(t) ∈ R3 is the translation and R(EM(t)) ∈ SO(3) is the rotation. We approximate M at every time t by using M at few times ti ∈ [0,1] corresponding to the key-frames provided by standard GS-multicamera structure-from-motion: M(t) is a linear combination of three M(ti) thanks to Taylor approximations and by neglecting their remainders. The y-th line of the j-th camera in the i-th keyframe is taken at time t = ti +∆ j + yτ . Following [2], a minimal parametrization R is preferred to avoid any constraints on the R entry and limit the number of estimated parameters: k = 3. However, R is a global parametrization of all rotations of the continuous camera motion and R has singularities like every 3D parametrization of SO(3). We propose to define R using a careful use of the Euler parametrization and keep away from the Euler singularities thanks to an assumption on the camera (all yaw angles are possible, but the pitch and roll are small). This assumption is reasonable for an user exploring the environment without a special objective like grasping at objects on the ground. We experiment in cases that we believe useFigure 1: Helmet-held multi-camera formed by four GoPro Hero3 cameras, images taken at a viewpoint, reconstruction of a 900m long video sequence (walking in a town) by RS-SFA bundle adjustment without loop closure.